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The  use  of  unmanned  aerial  systems  (UASs)  is  expanding  rapidly.  In  military  operations,  this 
increased  use  was  often  accompanied  by  relatively  high  mishap  rates  compared  with  rates  across 
more  mature  manned  aircraft.  These  higher  rates  led  to  multiple  high-level  reviews  of  unmanned 
operations  to  understand  the  issues,  but  surprisingly  little  consensus  emerged  across  reports 
regarding  root  causes.  To  help  close  this  gap,  Air  Force  Predator  Class  A  mishap  reports  through 
FY  2006  were  analyzed  in  detail.  Mishap  rates,  counts,  and  causal  factors  were  all  found  to  shift 
systematically  over  time,  with  a  rise  in  mishap  reports  citing  shortfalls  in  several  skill  and 
knowledge  areas  in  FY  2004-2006.  Individual  and  team  Predator  training  objectives  were  revisited 
and  the  content  of  crew  resource  management  (CRM)  training  was  refocused  on  improving  these 
key  operator  skills.  In  FY  2007-2008,  Predator  Class  A  mishaps  attributed  to  operator  error 
decreased  despite  increasing  numbers  of  mishaps  overall. 


While  early  attempts  to  use  UASs  for  military  purposes  can  be  traced  back  to  World  War  II  or  before 
(Gambone,  2002),  unmanned  technology  clearly  entered  the  mainstream  of  combat  operations  during  the  recent 
conflicts  in  Kosovo,  Afghanistan,  and  Iraq.  In  these  conflicts,  UASs  were  first  used  as  Intelligence,  Surveillance, 
and  Reconnaissance/Target  Acquisition  assets,  providing  commanders  with  imagery  intelligence,  electronic 
intelligence,  and  streaming  video.  Resulting  information  could  be  used  to  monitor  enemy  movements  and  conduct 
battle  damage  assessment.  The  Predator  system  added  a  strike  capability,  and  similar  capabilities  were  not  far 
behind  in  other  Department  of  Defense  (DoD)  UAVs.  Across  all  United  States  military  services,  the  use  of 
unmanned  aerial  systems  (UASs)  is  expanding  rapidly.  Predators  alone  already  account  for  5%  of  all  Air  Force 
flying  hours  in  FY  2008  (Air  Force  Safety  Center,  2009).  Despite  the  rapid  rise  of  flying  hours  to  date,  only  about 
one  half  of  requests  for  UAS  surveillance  can  currently  be  met,  with  growth  in  flying  hours  being  limited  by  the 
ability  to  train  enough  crews  to  meet  the  demand  for  battlefield  surveillance.  The  Air  Force  now  flies  27  round-the- 
clock  Predator  and  Reaper  orbits  in  the  Central  Command  area  of  operation,  which  involves  450  pilots.  Service 
leaders  want  50  orbits  to  be  flown  by  2012,  which  will  require  1,100  pilots  (Hoffman,  2008).  The  Quadrennial 
Defense  Review  predicted  that  approximately  45%  of  the  future  long-range  strike  force  will  be  unmanned  (Office  of 
the  Secretary  of  Defense,  2006).  Emerging  roles  for  UAVs  include  homeland  security  (e.g.,  border  patrol),  long- 
duration  law  enforcement  surveillance,  and  delivery  of  critical  medical  supplies  needed  on  the  battlefield  (Bone  and 
Balkcom,  2003). 

This  rapid  rise  in  UAV  employment  was  unfortunately  accompanied  by  high  mishap  numbers  across  all 
military  services.  This,  in  turn,  led  to  several  senior  reviews  to  understand  the  root  causes.  The  Office  of  the 
Secretary  of  Defense  published  a  report  on  UAV  reliability  in  2003)  that  looked  at  non-weather  related  mission 
aborts  or  cancellations.  The  “Achilles  heels”  of  UAV  platforms  appeared  to  revolve  around  component  quality, 
redundancy,  and  maintenance,  and  concluded  that  it  was  critical  to  improve  UAV  platforms  in  these  areas  because 
reliability  affects  affordability,  availability,  and  acceptance.  A  Defense  Science  Board  Study  on  Unmanned  Aerial 
vehicles  and  Uninhabited  combat  Aerial  Vehicles  (2004)  concluded  that  UAV  programs  have  not  yet  expended  the 
resources  necessary  to  fix  the  root  causes  leading  to  mishaps,  and  that  manned-aircraft-like  reliability  is  achievable, 
but  will  require  substantial  additional  investment.  Tvaryanas,  Thompson,  and  Constable  (2005)  conducted  an  in- 
depth  review  of  UAV  mishaps  across  the  United  States  military  services.  They  reported  that,  since  the  inception  of 
the  systems  in  the  1990s  through  the  end  of  FY  2003,  334  mishaps  per  100,000  flying  hours  had  occurred  with  the 
Navy/Marine  Corps  Pioneer,  55  mishaps  per  100,000  flying  hours  had  occurred  with  the  Army’s  Hunter  system,  and 
32  mishaps  per  100,000  flying  hours  occurred  with  the  Air  Force’s  Predator  system.  For  comparison  purposes, 
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overall  Air  Force  Class  A  mishap  rates  ($1  million  damage  or  fatality)  are  typically  in  the  low  single  digit  range  per 
100,000  flying  hours  (O’Toole,  Hughes,  &  Musselman,  2006). 


A  recent  challenge  from  the  Secretary  of  Defense  to  reduce  the  numbers  of  preventable  mishaps  by  at  least 
75%  (Rumsfeld,  2006)  focused  attention  on  Predator  mishap  frequencies  which  accounted  for  25%  of  all  Air  Force 
Class  A  AIB  reports  (at  least  $1  million  damage  or  a  fatality)  in  the  past  two  years  (FY  2006  and  2007).  It  should  be 
noted  that  most  manned  aircraft  are  mature  systems,  while  most  UAV  programs  are  relatively  early  in  their  life 
cycles,  and  mishap  rates  tend  to  improve  with  system  maturation. 

A  consistent  picture  of  the  problem  to  be  solved  has  not  yet  emerged  for  UAS  mishap  reduction.  Even  at 
basic  levels  such  as  the  relative  contributions  of  equipment  failure  versus  human  error,  different  analysts  reached 
widely  differing  conclusions.  The  Office  of  the  Secretary  of  Defense  Reliability  Study  (2003)  reported  that  human 
error  represented  16%  of  all  sources  of  Predator  A  (MQ-1)  system  failures  and  2%  of  Predator  B  (MQ-9)  mishaps 
and  the  Defense  Science  Board  (2004)  reported  that  17%  of  UAS  mishaps  were  attributable  to  human  error.  On  the 
other  hand,  Tvaryanas  and  his  colleagues  (2005)  reported  that  68%  of  UAS  mishaps  involved  causal  human  factors 
and  Williams  (2004)  reported  that  67%  of  Predator  mishaps  involve  human  factors.  Some  researchers  looked  at 
Class  A  (more  than  $1  million  damage  or  a  fatality),  B  (more  than  $500,000  damage,  and  C  (more  than  $20,000 
damage)  mishaps  (e.g.,  Tvaryanas,  2006),  some  considered  Class  A  mishaps  only  (Williams,  2004),  and  others  did 
not  specify  the  scope  of  the  mishaps  analyzed. 

Experience  with  previous  efforts  to  reduce  mishaps  in  manned  aircraft  dictates  that  successful  interventions 
to  improve  reliability  must  be  based  on  an  accurate  understanding  of  the  root  causes  leading  to  failure.  Several 
researchers  recently  documented  differing  root  cause  patterns  across  organizations  and  platforms.  Helmreich, 
Wilhelm,  Klinect,  and  Merritt  (2001)  studied  threats  to  safety  and  the  nature  of  errors  in  three  domestic  air  carriers 
in  the  United  States,  and  observed  striking  differences  among  these  airlines  regarding  both  threats  to  safety  and 
operator  errors  despite  obvious  commonality  with  respect  to  mission  and  environment.  Nullmeyer,  Stella,  Montijo, 
and  Harden  (2005)  reported  differing  mishap  root  causes  across  Air  Force  manned  aircraft  types.  Williams  (2004) 
reported  major  deviations  in  root  causes  across  UASs,  and  Tvaryanas,  et  al.  reported  significant  differences  among 
root  causes  depending  on  the  service  involved. 

Based  on  rapidly  increasing  UAS  operations  in  both  military  and  civilian  organizations,  the  emphasis  from 
senior  military  leaders  on  reducing  UAS  mishaps,  and  the  lack  of  consensus  in  the  literature  on  causal  factors,  we 
felt  that  root  cause  analyses  with  known  parameters  were  needed  to  assess  the  role  that  training  interventions  could 
play  to  reduce  mishaps  and  increase  capability  for  a  given  platform.  Our  focus  in  this  paper  is  on  root  causes  and 
other  characteristics  of  Air  Force  Predator  Class  A  mishaps.  This  focus  was  chosen  in  part  because  Class  A  mishap 
reports  are  more  detailed  than  Class  B  or  Class  C  mishap  reports,  and  in  part  because  Class  A  mishap  counts  have 
become  a  highly  visible  metric  of  safety  and  reliability.  Based  on  the  patterns  that  emerged  from  our  analyses, 
training  interventions  are  proposed  to  address  the  areas  of  greatest  potential. 

Nullmeyer,  Herz,  Montijo  and  Leonik  (2007)  analyzed  findings  from  all  Air  Force  Predator  Class  A 
mishaps  that  had  occurred  from  the  introduction  of  this  system  into  the  Air  Force  inventory  in  1995  through  the  end 
of  FY  2006  to  identify  training-related  trends.  Substantial  changes  were  reported  over  time  regarding  annual  mishap 
rates,  annual  mishap  counts,  and  causal  factors.  Mishap  rates  across  the  past  three  years  were  consistently  less  than 
one  half  the  combined  rate  across  earlier  years.  Mishap  counts ,  however,  steadily  increased,  as  did  Predator  flying 
hours.  Early  mishap  reports  typically  cited  mechanical  problems  and  operator  station  design  issues.  From  2003 
through  2006,  80%  of  mishaps  cited  causal  human  error  factors.  Equipment  interface  problems  were  still  cited  as 
causal  or  major  contributing  factors  in  almost  half  of  these  mishaps.  More  specifically,  mishap  reports  from  2003- 
2006  often  cited  shortfalls  in  skill  and  knowledge  (checklist  error,  task  misprioritization,  lack  of  training  for  task 
attempted,  and  inadequate  system  knowledge),  situation  awareness  (channelized  attention),  and  crew  coordination. 

Based  on  the  findings  of  Nullmeyer,  et  al.,  crew  resource  management  training  was  developed  for  both  the 
Predator  formal  school  and  for  continuing  Crew  Resource  Management  training  that  was  given  to  mission  qualified 
crews.  The  focus  of  the  new  courseware  was  having  students  understand  the  primary  threats  to  safety  in  the  Predator 
community  and  providing  techniques  to  manage  the  types  of  operator  error  that  were  repeatedly  cited  in  Predator 
mishap  reports.  The  remainder  of  this  paper  updates  previous  findings  regarding  human  factors  trends  in  Predator 
mishaps,  focusing  on  publicly  accessible  information. 


Methods 


The  United  States  Air  Force  Judge  Advocate  General’s  office  maintains  an  online  repository  of  Accident 
Investigation  Board  (AIB)  report  summaries  for  Air  Force  Class  A  mishaps.  This  site  (http://usaf.aib.law.af.mil)  is 
publicly  accessible,  lists  Class  A  mishaps  by  fiscal  year  across  all  platforms  in  the  Air  Force,  and  provides  one  page 
executive  summaries  of  AIB  reports  as  they  are  released.  These  summaries  describe  the  mishap  and  discuss  probable 
cause.  Most,  but  not  all  Class  A  mishaps  are  analyzed  by  an  AIB.  The  publicly  accessible  database  accounted  for 
over  90%  of  all  Class  A  Predator  mishaps. 

The  Air  Force  Safety  Center  generates  mishap  investigation  reports  for  every  Class  A  mishap  and  provides 
results  at  varying  levels  of  granularity.  The  analyses  reported  here  provided  the  structure  for  further  analyses  of 
information  from  three  distinct  Safety  Center  sources  that  were  used  to  guide  changes  in  mishap  reduction  training. 
Moving  from  general  to  specific,  the  first  was  statistical  data  from  the  Air  Force  Safety  Center  web  site 
(http://afsafety.af.mil).  These  data  include  hours  flown  and  numbers  of  Class  A  mishaps  by  fiscal  year  and  by 
aircraft  type.  The  second  data  source  was  safety  investigation  summaries.  These  provide  a  brief  narrative  of  the 
mishap,  and  categorized  the  Predator  Class  A  mishaps  as  being  primarily  logistics-,  maintenance-,  or  operations- 
related.  Summaries  also  provide  descriptive  data  for  each  mishap  such  as  phase  of  the  mission  and  time  of  day  in 
which  the  mishap  occurred,  and  list  conclusions  and  recommendations.  The  third  Safety  Center  source  used  was 
discussions  of  human  factors  from  the  full  mishap  investigation  reports.  Safety  Investigation  Board  (SIB)  findings 
are  formally  documented  as  a  section  of  the  full  mishap  report.  These  findings  were  reviewed  for  descriptions  of 
human  factors  causing  or  contributing  to  the  mishap.  In  addition  to  the  board  findings,  a  separate  Life  Sciences 
Report  is  prepared  by  the  Life  Sciences  Branch  of  the  Air  Force  Safety  Center.  The  Life  Sciences  Report  provides  a 
chronological  mishap  narrative  and  a  discussion  of  every  element  cited  in  the  human  factors  database. 
Interrelationships  among  the  human  factors  may  be  addressed. 

AIB  summary  reports  (http://usaf.aib.law.af.mil)  were  initially  analyzed  to  generate  descriptive  trend  data 
regarding  mishap  frequencies  and  the  general  nature  of  the  mishaps  (equipment  failure  or  operator  problems)  over 
time.  Flying  hour  data  were  obtained  from  the  Air  Force  Safety  Center  web  site  (http://afsafety.af.mil).  Data  from 
both  sources  were  combined  to  generate  mishap  rates. 

Four  training-related  problem  areas  emerged  from  analyses  of  the  full  and  more  detailed  Safety  Center 
mishap  reports:  (1)  situation  awareness  development  and  maintenance,  (2)  task  management,  (3)  decision  making, 
and  (4)  crew  coordination.  Legacy  Crew  Resource  Management  (CRM)  training  for  both  initial  qualification 
training  students  and  recurrent  training  for  qualified  operators  was  refocused  on  these  four  areas  and  on  Predator 
operations.  This  new  training  continued  to  be  accomplished  in  a  seminal  format,  but  content  and  case  studies  were 
updated  to  emphasize  Predator-specific  threats  to  safety  that  are  under  the  control  of  crews  and  strategies  to  mitigate 
the  types  of  crew  error  that  have  led  to  Class  A  mishaps. 

Results 

Baseline  Predator  mishap  frequencies  increased  systematically  over  time  as  shown  in  Figure  1 
(http://usaf.aib.law.af.mih  2009).  Fiscal  years  accounted  for  over  65%  of  the  variability  observed  in  mishap 
frequencies  (correlation  =  .81,  p  <  .001).  Causes  in  AIB  reports  are  most  usually  stated  in  terms  of  a  single  “clear 
and  convincing”  cause.  As  part  of  our  analyses,  this  cause  was  categorized  as  being  either  operator  error  or  some 
other  factor,  usually  equipment  failure.  The  primary  cause  of  the  mishaps  appeared  to  be  shifting  toward  operator 
error  problems  over  time  through  FY  2006.  Three  Class  A  Predator  mishaps  that  predated  the  AIB  databases.  Two 
of  these  three  were  attributed  somewhat  evenly  between  equipment  failure  and  operator  error.  Mishaps  in  the  early 
years  (FY  1998  -  2003)  were  most  often  attributed  to  equipment  failure.  Mishaps  from  the  next  three  years  (FY  2004 
-  2006)  were  attributed  primarily  to  operator  error. 

Safety  Center  Class  A  reports  provide  more  detailed  descriptions  that  address  multiple  causal  and  contributing 
factors  and  the  interactions  among  them.  In  the  detailed  Safety  Center  analyses,  nine  of  15  mishaps  from  FY  1997- 
2003  were  attributed  to  equipment  factors,  and  even  four  of  the  six  operator-error  mishaps  cited  causal  equipment 
interface  problems.  In  total,  thirteen  of  the  fifteen  mishaps  from  1997  through  2003  cited  causal  equipment  factors. 
In  FY  2003-2006,  one  mishap  was  attributed  primarily  to  equipment  failure  and  the  remaining  14  were  attributed  to 
operator  (12  mishaps)  or  maintainer  (2  mishaps)  error.  Further,  only  three  of  the  12  mishaps  attributed  to  operations 


Figure  1:  Predator  Class  A  Mishap  Frequencies  by  Fiscal  Year  (http://usaf.aib.af.mil) 
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cited  equipment  as  a  causal  factor.  Functional  design  continued  to  be  cited  as  a  contributing  factor,  however,  in 
many  of  these  recent  mishaps. 

Mishap  frequencies  after  training  modifications  that  were  introduced  at  the  end  of  FY  2006  continued  to  rise  in  FY 
2007  and  2008,  Twenty  six  mishaps  occurred  between  2000  and  2006,  averaging  3.7  per  year.  Sixteen  additional 
mishaps  occurred  between  2007  and  2008,  averaging  8  per  year.  The  AIB  database  revealed  a  substantial  drop  in 
mishaps  attributed  to  operator  error  and  reflected  a  reversal  of  probable  cause  from  operator  error  through  2006  to 
other  factors,  almost  always  equipment  failure,  since  then.  A  similar  pattern  was  seen  in  Safety  Center  mishap 
reports  (chi  square  =  7.61,  df  =  1,  p  <  .01). 

Predator  flying  hours  are  shown  in  Figure  2.  The  numbers  of  Predator  mishaps  clearly  need  to  be 
interpreted  in  the  context  of  the  accelerating  growth  of  flying  hours  are  reported  on  the  Air  Force  Safety  Center  Web 
site  (http://afsafety.af.mil).  Annual  flying  hours  increased  from  less  than  3000  in  FY  2000  to  almost  80,000  in  FY 
2007,  the  latest  year  reported.  Projections  call  for  continuing  increases  in  UAS  operations.  These  changing 
utilization  levels  are  important  to  consider  when  interpreting  trends  in  mishap  frequencies  over  time. 


Figure  2.  Predator  Hours  Flown  Annually  from  2000  -  2007 


Fiscal  Year 


Mishap  rates  per  100,000  flying  hours  help  take  into  account  the  rapidly  growing  utilization  of  Predators  and 
provide  a  better  metric  of  safety  then  frequencies.  Figure  3  depicts  both  Predator  and  historic  F-16  mishap  rates 
starting  with  the  first  year  in  which  more  than  5000  hours  were  flown  annually  in  either  platform.  With  rapidly 
increasing  operations,  mishaps  per  100,000  flying  hours  decreased  substantially  since  the  early  2000’s  despite 


increasing  mishap  counts.  Comments  from  some  of  the  early  reviewers  suggesting  unusually  high  mishap  rates  in 
UASs  were  based  on  the  early  years,  when  mishap  rates  were  high.  Over  time,  Predator  mishap  rates  are  following  a 
pattern  that  is  very  similar  to  the  rates  seen  early  in  the  history  of  the  F-16  weapon  system.  Both  systems 
encountered  mechanical  and  human  error  problems  early  in  their  life  cycles.  F-16  mishaps  are  now  very  close  to 
overall  Air  Force  mishap  rates.  For  F-16  mishaps,  the  data  points  represent  the  time  period  between  1977  and  1984. 
Predator  Class  A  mishap  rates  were  about  6  per  100,000  flying  hours  in  2007  and  are  projected  to  be  close  to  10  per 
100,000  flying  hours  in  FY  2008.  For  comparison  purposes,  the  overall  Air  Force  Class  A  mishap  rate  has  been 
slightly  less  than  two  mishaps  per  100,000  flying  hours  for  the  past  decade. 

Figure  3:  RQ-1  (Predator)  and  F-16  Class  A  Mishap  Rates  (with  Historic  F-16  Rates  FY  77-85) 


Conclusions 

Predator  mishap  trends  reflected  systematic  and  substantial  changes  over  time.  The  overall  direction  of 
these  trends  depends  on  the  measure  used.  Mishap  frequencies  steadily  increased  over  time  as  have  Predator  hours 
flown.  Mishap  rates  per  100,000  flying  hours  decreased  substantially  (from  23  Class  A  mishaps  per  100,000  flying 
hours  from  fiscal  years  1997-2003  to  less  than  1 1  in  fiscal  years  2004-2007).  Despite  the  decrease,  Predator  mishap 
rates  remain  high  relative  to  more  mature  Air  Force  weapon  systems,  but  they  are  similar  to  the  rates  seen  in  the 
early  years  of  F-16  operations  and  are  dropping  quickly. 

In  FY  2004-2006,  a  substantial  increase  was  observed  in  mishaps  that  cited  insufficient  operator  skills  and 
knowledge.  The  threat  and  error  management  model  (Helmreich,  et  al,  2001)  is  widely  used  by  air  carriers  to 
enhance  safety.  We  believe  that  it  also  provides  a  reasonable  structure  for  improving  UAV  mishap  rates  in  military 
operations  and  ultimately  for  increasing  combat  capability.  A  key  part  of  this  approach  is  to  use  evidence  to 
structure  interventions  to  alleviate  the  specific  problems  that  actually  plague  a  particular  community.  Training  is 
one  of  several  tools  that  can  be  used  to  meet  safety  and  capability  objectives,  but  other  changes  such  as  equipment 
modifications  and  altered  procedures  may  also  be  integral  parts  of  an  effective  overall  error  mitigation  strategy.  The 
bottom  line  is  that  the  better  we  understand  the  real  threats  to  safety,  the  more  successful  we  are  likely  be  in 
developing  effective  strategies  to  mitigate  them.  With  the  recent  rise  in  equipment-related  problems,  it  would  be 
prudent  to  address  the  role  of  human  error  in  equipment  maintenance  and  if  warranted,  develop  threat  and  error 
management  training  for  maintainers.  Following  the  effort  to  refocus  CRM  training  to  address  known  threats  to 
safety,  overall  mishap  rates  continued  to  climb,  but  the  causes  cited  in  AIB  reports  shifted  from  human  error  to 
equipment  failure.  Similar  patterns  were  apparent  in  the  more  detailed  Safety  Center  mishap  reports. 

To  substantially  increase  the  numbers  of  Predator  operators,  the  Air  Force  is  currently  evaluating 
alternatives  to  using  experienced  pilots  to  control  Predator  platforms.  Two  programs  are  underway,  one  looking  at 
the  ability  of  pilots  who  recently  completed  undergraduate  pilot  training  and  another  assessing  the  ability  of  non¬ 
pilot  candidates  to  perform  the  tasks  required  of  Predator  pilots.  One  measure  of  merit  is  mishap  rates.  It  is  clear  that 


mishap  frequencies,  rates  and  causes  are  all  dynamic  in  the  emerging  field  of  UAS  operations,  and  that  mishap 
reports  provide  a  fertile  source  of  insight  into  where  training  and  operations  need  to  be  improved.  Our  analyses 
suggest  that  raw  frequencies  could  be  misleading,  especially  in  light  of  few  operator  error  mishaps  in  the  past  two 
years.  Instead,  safety  analyses  for  the  purpose  of  assessing  crew  performance  need  to  focus  on  mishaps  where 
operator  error  is  a  factor. 
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